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Abstract: Process evaluation helps us to understand the 
planning process. This predominantly qualitative approach 
explains how and why decisions are made and activities 
undertaken. The focus includes feelings and perceptions of 
program staff. The evaluator's ability to interpret and 
longitudinally summarize the experience of program staff and 
community members is critical. Techniques discussed 
include participant observation, content analysis, situational 
analysis, in-house surveys, and interviews. By combining 
sources and methods, a fuller picture of the process is 
revealed. 


What exactly is process evaluation? Is it really evaluation at all? 
The answers to these questions may be less straightforward than the 
questions themselves. Process evaluation, as an emerging area of evaluation 
research, is generally associated with qualitative research methods, though 
one might argue that a quantitative approach, as will be discussed, can also 
yield important insights. 

We offer this definition of process evaluation developed by the Federal 
Bureau of J ustice Administration : 1 

Process Evaluation focuses on how a program was 
implemented and operates. It identifies the procedures 
undertaken and the decisions made in developing the 
program. It describes how the program operates, the 
services it delivers, and the functions it carries out . . . 
However, by additionally documenting the program's 
development and operation, process evaluation assesses 
reasons for successful or unsuccessful performance, and 
provides information for potential replication [ italics added]. 
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The last sentence in this definition is at the heart of process 
evaluation's importance for Circles of Care (CoC). Process evaluation is a 
tool for recording and documenting salient ideas, concerns, activities, 
administrative and management structures, staffing patterns, products, and 
resources that emerge during three-year CoC planning grants. Unlike outcome 
evaluation, which often measures the results of a project's implementation 
against its programmatic projections, there are not necessarily a priori 
assumptions about what the planning process will look like. 

Furthermore, as discussed in an earlier chapter on the life cycle of 
the evaluation process, there are stage-specific developmental activities 
occurring within the program. While the specific context will vary across 
projects, we may assume that there are common dynamics (e.g., Process, 
Development and Action Stages) that when understood can frame the 
experience and be helpful to participants and next generation planners. 

I n essence, process evaluation entails tracing the footsteps that CoC 
staff, as well as others involved in planning activities, have taken in order to 
understand the paths that have been traveled, as well as journeys started 
and later abandoned. This process is akin to the grounded theory approach 
of qualitative evaluation (Artinian, 1988; Strauss & Corbin, 1990). Process 
evaluation is an inductive method of theory construction, whereby observation 
can lead to identifying "strengths and weaknesses in program processes and 
recommending needed improvements" (Rubin & Babbie, 2001, p. 584). 

To better understand process evaluation aligned with the qualitative 
tradition, we borrow from Rubin and Babbie (1993) for an operational definition 
of qualitative methods: 

Research methods that emphasize depth of understanding, 
that attempt to tap the deeper meaning of human 
experience, and that intend to generate theoretically richer, 
observations which are not easily reduced to numbers are 
generally termed qualitative methods, (p. 30). 


We deduce from this definition the evaluator's unique role as the 
tool that synthesizes the "human (collective) experience” of CoC participants. 
Regardless of methods - participant or direct observation, unstructured or 
intensive interviewing - it is the evaluator who ultimately classifies, 
aggregates, or disaggregates themes that emerge as a result of the planning 
process. 

As has been discussed elsewhere in this Special Issue, the evaluator's 
relationship with the CoC team is an integral part of the evaluation. It is 
especially paramount with regard to process evaluation, given the relative 
intimacy of interaction required by some of the data collection techniques. 
As may be expected, this "at your side” approach can intensify strained or 
suspicious relationships between the evaluator and program staff. 
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As one CoC program staff member explains: 

When I think about these terms ‘qualitative research' and 
'participant observer/ 1 feel the abusive history of my people 
staring me in the face. I ntense feelings of anger, hurt, and 
betrayal all come into play. Being in a fish bowl comes to 
mind, as do memories of ‘tourists' who visited the 'mission,' 
which stood on my reservation, and took pictures of the 
'I ndian children,' and made comments like 'how poor' and 
'uncivilized' we were. 


As I understand the term 'participant observation,' I feel 
insulted. Feelings of betrayal, falsehoods, and sacrilege 
come to mind. Our culture and our way of processing is 
who we are as a people. It is all very intimate in nature. In 
Circles of Care we trusted to open ourselves up, to share 
ourselves, our culture, and to take the time to know those 
who were not of our culture. This was a big step and not 
one taken lightly. Knowing that someone participated as 
one of us, yet in turn dissects the process, is not being true. 


Process evaluation thus requires vigilance on the part of the evaluator 
to respect the trust that has been afforded him or her by American I ndian 
and Alaska Native (AI/AN) program staff. The evaluator's observations and 
comments should be made knowing that there are cultural and historic 
overtones and undercurrents that influence the interpretation of events, as 
well as the meaning that CoC program staff assign to the process evaluation 
description. Process evaluation, just like any other form of assessment, 
requires cultural sensitivity and awareness. It may be that certain techniques 
(e.g., participant observation) are not appropriate tools for evaluators that 
enter a program without prior relationships with the CoC program staff. 

Having addressed at the onset the evaluator's role in process 
assessment, and mindful that working relationships will evolve during the 
life cycle of the project, the evaluator is ready to engage in the process 
evaluation. There are several conventional evaluation techniques that can 
be used to discern and describe the CoC planning process itself. They are: 
participant observation, content analysis, situational analysis, in-house 
surveys, and interviews. This multi-source approach is consistent with Marcus' 
( 1988) recommendation that the collection of official documentation should 
be combined with the input of "key actors.” Strauss and Corbin ( 1990) also 
support this approach by advocating for qualitative data collection from a 
grounded theory perspective. They point to the emergence of a 
representativeness of concepts, which is to say that themes can be generalized 
based on the similarities across the phenomena being studied. 
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With the exception of in-house surveys, these techniques are 
qualitative in nature, suggesting that Rubin and Babbie's (1993) definition of 
qualitative research's focus on understanding and the deeper meaning of 
human experience is most apt in the process evaluation domain. I n his or 
her approach to qualitative assessment, the evaluator is interested in 
understanding the content and meaning of written and oral expressions. 
One helpful approach is to assess content based on manifest and latent 
themes (Rubin & Babbie, 1993). Manifest content refers to the frequency 
that certain words, phrases, or concepts appear in documents and oral 
expressions, such as recurring themes of specific resource needs and their 
sources, expressions of feelings (e.g., tired, excited, or fulfilled), categories 
of persons targeted for involvement as informants, or the kind of technical 
assistance requested. Latent level analysis entails the evaluator's overall 
assessment of the project activities or concerns, the input, its clarity of purpose 
and direction, and current level of development. 

Process Evaluation Techniques 

The following is a discussion of process evaluation techniques that 
are used by CoC grantees. 

Participant Observation 

Though there is a range of participatory roles that evaluators can 
play that run the gamut from fully immersed and invisible participant to fly on 
the wall sidelines observer, the common experience of CoC grantees is to 
have the evaluator in the observer-as-participant role (Gold, 1969). In this 
capacity, the evaluator's responsibilities and duties are clearly known to the 
planning and program staff, and to community members. There is no attempt 
to disguise the evaluator's role. Credibility and trust are of utmost importance. 

Evaluators are present at key planning meetings involving CoC staff 
and community agencies. They listen at focus group sessions with families 
and youth, attend Gathering of Native Americans (GONA) events and 
community picnics, join in progress presentations to sponsoring agency boards 
of directors, and attend regional and national meetings with other staff 
members. When evaluators' roles are among the reasons for their 
participation, they fulfill these responsibilities by developing surveys, discussing 
data collection strategies, and reporting results. Regardless, however, of 
these assigned duties, evaluators also reflect on the content of each event, 
and attempt to categorize elements into thematic and descriptive domains. 
One evaluator's reflections are provided below: 
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The GONA provided important insights and a rich contextual 
understanding of tribal and community perspectives for 
participants. Several workgroups were formed during the 
GONA that were charged with identifying community 
strengths and needs, and participants provided examples 
from their personal experiences. The GONA experience, 
occurring within one year of the project's initiation, seems 
to have added new vitality and clarity about the project's 
purpose, and has increased support for the initiative among 
community leaders. 

While participant observation is a primary source for uncovering 
themes and obtaining richer understanding of the process' context, secondary 
sources, such as content analysis, can be equally as informative. 


Content Analysis 

Content analysis refers to a systematic review of written documents 
produced by CoC staff, volunteers, and community members. I ncluded are 
planning documents such as timelines, resource lists, and budgets, promotional 
materials such as flyers, letters to allied agencies and others explaining the 
initiative, minutes of meetings, proposals for funding and applications for 
special recognitions, as well as any other documents that capture features 
of the project. 

Content analysis focuses on the ideas being communicated. With 
the evaluator as the instrument for assessing the content of written materials, 
he or she lists or codes ideas, words, and phrases that capture salient elements 
of the program. Since the process evaluation has a longitudinal perspective 
(e.g., what issues, concerns, and strategies characterize the project at a 
given point in time), it is also necessary to note the temporal sequencing of 
events and to be clear about the units of analysis, which are the planning 
team and community members. Maintaining a macro focus is essential for 
content analysis to be helpful in supporting the process evaluation in that the 
inquiry pertains to replicable actions and stages, as well as any activities that 
have not proven to be productive or helpful. 

One example of content analysis is a review of reports from 
newsletters produced by Feather River Tribal Health on their sponsorship of 
community picnics as a tool for community organizing and building awareness 
of resources for families. The first community picnic was held October 1, 
1999. Below is the description of the event in the project's newsletter: 
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At our Wellness Retreat (GONA) last June, one message 
rang loud and clear: community togetherness. In these 
fast moving times of change, getting together with family, 
friends, and community members is often difficult to fit into 
one's schedule. The true community gatherings have 
become a thing of the past, becoming something we 
remember as "when I was younger . . ." Held at Durham 
Park, with a turn out of approximately 140 people . . . 


A newsletter report on the second community picnic said: 

On April 22, 2000, the Circles of Care staff . . . joined forces 
to put together a community picnic at Palermo Park. 
Approximately 350 adults and children attended this event . 

. . The heat from the grill was hot, but the day was cool and 
feelings of community belonging ran strong . . . Community 
events such as these are important to building and 
maintaining relationships within the community. I n a time 
when it is easy to lose touch with family and loved ones, 
family and friend get-togethers are a perfect way of slowing 
down and touching base. 


Aside from the large increase in the number of attendees between 
events, which in and of itself is instructive about the promotion and draw 
that such events can achieve, we observe manifest terms such as "community" 
and "family," and latent themes characterized by "slowing down” and "touching 
base." Values similar to those communicated in the newsletter emerged in 
other contexts, as well, as at provider meetings and focus group sessions, 
thus validating their importance to the project. 


Situational Analysis 

A technique that has been utilized by some projects to assess the 
CoC planning process involves periodic meetings of the planning staff facilitated 
by the evaluator to conduct what is referred to as a SWOT analysis. SWOT 
is an acronym standing for strengths, weaknesses, opportunities and threats 
(Barry, 1986). In a SWOT analysis, the following questions are sequentially 
asked: 
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1. What are our organization's internal and external strengths? 

2. What are our organization's internal and external weaknesses? 

3. What opportunities are present, within our organization and in the 
environment that surrounds us? 

4. What threats exist - either internally or externally - that could adversely 
affect our efforts? 


The SWOT assessment is focused on both internal and external 
dynamics. Thus, the focus is divided between organization concerns, and 
community and environmental issues. Combined, these factors represent a 
full spectrum of issues and concerns facing the project at any given point in 
time. Utilizing this approach it is possible to longitudinally observe changes 
in staff perceptions as the planning effort evolves. 

For example, a SWOT assessment conducted at an early stage of a 
CoC grant reveals strength - based values such as teamwork, ability to network 
across cultures, and flexibility in program design and use of funds. Weaknesses 
pertain to misunderstandings about planning among segments of the 
community, lack of credibility as a planning entity, and concern that internal 
support is not adequate. Opportunities at the early stage relate to 
complementary funding streams, new dialogues with public service providers, 
and descriptive data on resources and service levels that heretofore had not 
been available. Finally, threats relate to perceptions that CoC will be seen as 
a temporary project without lasting impact, and concern that public agencies, 
having low cultural competency in serving AI/ANs, will resent challenges to 
their methods of intervention. 

During the third and final year of the planning project, the SWOT 
reveals a different set of perceptions (see Table 1). Monitoring changes in 
perception through triangulation of sources, such as in-house survey discussed 
below, again enriches understanding and strengthens conclusions. 


I n- House Surveys 

Though we have characterized process evaluation as primarily a 
qualitative method, there are also opportunities to inform the process 
evaluation by relying on quantitative tools. For example, a variation of the 
situational analysis can be performed vis-a-vis an in-house survey, whereby 
staff and community members are asked to report their awareness of and 
satisfaction with the project at different points in time. A Likert-type scale 
can be developed, whereby respondents rate such variables as community 
awareness, provider cooperation, management support, timeline pace, or 
adequacy of resources on an ordinal scale ranging from "l"to "5". 

This approach can provide input into the planning process from a 
wider circle of respondents than core staff. It also provides a modicum of 
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anonymity for respondents, and allows those less reticent to be heard with 
equal voice. A mean score (average of responses) can be used to determine 
where the program is weak and strong, and where threats and opportunities 
are present. 


Table 1 

SWOT Assesment During the Third and Final Planning Year 


Strengths 

Weaknesses 

Opportunities 

Threats 

Getting support 
from administration 
and medical 

"Crankiness" - 
challenge to be 
seen as a 
professional when 
not feeling like one 

Numerous funding 
opportunities 

Potential loss of 
independence now 
that project is 
permanent and 
service focused. 

Getting ready for 
transition (moving) 

Transition to new 
facility - staff is 
uncertain about 
what changes will 
be made and how 
they will be 
affected by them 

Classes available on 
computer training for 
Family Resource 
Center 

Growth competition 
for limited space in 
new building. 

Funding request 
submitted for 
outreach for 
workers, trainers, 
and others. 

Long-term funding 
not identified 

Family Resource 
Center regional 
conferences 

Not enough staff to 
serve volume of 
anticipated clients 


For example the following is an excerpt from a report on a CoC 
planning effort in which an evaluator conducted a survey of COC program 
staff to measure their assessment of the planning process to date: 

Using a five-point scale (l=Never; 2=Seldom; 3=Usually; 
4=Frequently; and 5=Always), respondents were asked to 
rate the frequency at which they experience certain issues 
associated with the CoC program. See Table 2. 
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According to the survey, staff respondents reported that 
cooperation among community providers in data provision 
and activity promotion was low (mean = 1.91) - approaching 
the seldom level. They too indicate that there is presently 
modest community awareness (ranging somewhere 
between seldom and usually) of CoC's purpose (mean = 
2.35). These responses suggest weaknesses in and threats 
to the program if not addressed. On the other hand, 
respondents identified three relative strengths and 
opportunities for the program. They reported that they 
usually felt that there was management support of CoC 
(mean = 2.97), and that the program's development was 
on track (mean = 3.27). Finally, they indicated that resources 
were quite adequate for the planning effort (mean = 3.82). 

1 1 is sometimes possible to use these quantitative findings 
as a jumping off point to discuss with the planning staff the 
meaning of their aggregated responses. I nterviewing is an 
effective technique for developing narrative commentary to 
accompany quantitative ratings. 

Table 2 

Circles of Care Concerns ( N=36 ) 


Issue 

Mean 

Experience 


Score 

Range 

Cooperation among community providers in 
providing data and promoting activities. 

1.91 

Seldom 

Community awareness of Circles of Care's purpose. 

2.35 

Seldom 

Management support of Circles of Care. 

2.97 

Usually 

Circles of Care is where we should be at this point in 
its development. 

3.27 

Usually 

Resources are adequate to support the planning effort. 

3.82 

Frequently 


I nterviewing 

To better understand the CoC process, evaluators employ different 
forms of interviewing. Patton (1990) identifies three forms of qualitative, 
open-ended interviews. They are: (a) informal conversational interviews, 
(b) general interviews using a guided approach, and (c) standardized opened- 
ended interviews. The first approach, informal conversational interviews, 
seems to work well as a process evaluation method. 
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Rubin and Babbie (2001) characterize informal conversational 
interviews as an "unplanned and unanticipated interaction between an 
interviewer and a respondent that occurs naturally during the course of field- 
work observation" (p. 404). This definition implies that spontaneity is 
important. The evaluator's focus is on maximizing his or her understanding 
of events in real time, by asking participants what they think about a situation 
as it unfolds. At the end of planning or finance meetings, after a long day of 
GONA activities, or following a special event like a community picnic, the 
evaluator can debrief with staff and other participants as to whether their 
expectations were met, and in what ways. The evaluator can inquire about 
the event's meaning to them as planners and as community members. 

Though the activities and events will vary across the three years of 
planning, the evaluator's inquiry is consistent. He or she wants to know how 
the event or activity affects the planning process. Mental notes are made of 
each conversation that later are recorded. There is nothing secretive about 
this approach. The evaluator's role is known, and thus his or her questions 
are understood to be a part of the process evaluation. 

A sample dialogue between the evaluator and a CoC staff member 
might go something like this: 

EVAL : How do you feel about the Community Picnic now 
that it is over? 


STAFF : I 'm glad that we had so many people. I'd estimate 
more than 300. 


EVAL : What did you hope to accomplish? 

STAFF : Well we wanted to involve families that we hadn't 
seen before and to link them with community agencies. 


EVAL : How do you think that it went? Were there new 
families and did the linkages with agencies work out? 


STAFF : On the whole I think that we were successful. I 
wish that there had been additional agencies, especially 
youth services providers. Well need to reach out to them 
more aggressively next time. 

As you can see, the dialogue is reflective and informal, and attempts 
are made to capture the experience through the eyes of a CoC staff member. 
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Conclusions 

As we have seen, process evaluation is another tool on the evaluator's 
workbench. Like other evaluation techniques, it has its own specialized 
approaches, which are predominantly qualitative in nature. Like other 
evaluative endeavors, the evaluator's credibility with the CoC staff is an 
important part of the working relationship. The evaluator's ability to 
understand and summarize the experience of staff and community members 
is affected by their willingness to share their concerns and to trust the 
assessment process. 

Since process evaluation utilizes a formative analytical approach, its 
benefits accrue to the CoC program itself in addition to new grantees. For 
example, in one CoC grantee program, manifest themes that emerged from 
the SWOT process, such as the compilation of rich descriptive data on the 
Native American community emerged as a strength, following its earlier 
identification as a weakness due to its previous lacking. The descriptive data 
also was seen as an opportunity to educate non-Native providers about the 
community through development of a PowerPoint presentation that was 
delivered to policymakers and human services administrators. Sole source 
contracts for mental health services to the CoC host agency that followed 
these encounters are attributable to its germination during the SWOT process. 

A second example is the role that an evaluator of a CoC grant played 
as a participant observer. She worked with staff to develop a funding proposal 
for submission to a private foundation. Thorough documentation of the 
planning process that led to the project proposal was of special interest to 
the funder, which she drafted. This level of detail also portrayed the 
organization's capacity to continue to document its developmental processes 
as the proposed project unfolded. I n essence, the evaluator, as participant 
observer, likely enhanced the content of the proposal, while also helping 
project staff to crystallize its implementation plan. 

We have seen that participant observation, content analysis, 
situational analysis, in-house surveys, and interviews are each helpful 
techniques for conducting a process evaluation. Their usage, however, will 
depend on the evaluator's style and preferences. As we have shown, by 
combining data sources and methods, a more complete picture of the process 
evaluation emerges. 


Gary Bess, Ph.D. 
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http://www.bja.evaluationwebsite.Org/html/glossary/p.html 



